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Abstract — We consider transmission over a binary-input addi- 
tive wliite Gaussian noise channel using low-density parity-clieck 
codes. One of the most popular techniques for decoding low- 
density parity-check codes is the linear programming decoder. 
In general, the linear programming decoder is suboptimal. I.e., 
the word error rate is higher than the optimal, maximum a 
posteriori decoder. 

In this paper we present a systematic approach to enhance the 
linear program decoder. More precisely, in the cases where the 
linear program outputs a fractional solution, we give a simple 
algorithm to identify frustrated cycles which cause the output of 
the linear program to be fractional. Then adding these cycles, 
adaptively to the basic linear program, we show improved word 
error rate performance. 

I. Introduction 

We consider transmission over a binary-input additive white 
Gaussian noise channel (BIAWGNC) using low-density parity- 
check (LDPC) codes. The two most fundamental decoders in 
this context are the belief propagation (BP) decoder [1] and 
the linear programming (LP) decoder [2]. In this paper we are 
interested in the performance of the LP decoder There is an 
extensive literature on analysis and design of the LP decoder 
' for LDPC codes [2]-[5]. As is well known, LP decoders have 
the advantage that they provide the ML certificate. This means 
that, if the LP decoder outputs an integer solution, then it must 
be the maximum likelihood (ML) codeword. Thus in this case 
the LP behaves as an optimal decoder One can also say that 
in this case there is no duality gap. 

However, it is also known that in general the LP decoder is 
suboptimal [2]. I.e., there exists channel noise realizations such 
that the LP decoder outputs a fractional solution, known as 
pseudocodewords [4], but still there exists a unique codeword 
which minimizes the objective function. This implies that the 
LP decoder is not successful in finding the ML codeword. 
As a result, there is a gap between the performance of the 
LP decoder and the ML decoder Hence it is an interesting 
question to understand what causes the LP decoder to fail and 
further if there exists methods to improve the LP decoder It is 
well known that adding redundant parity-check nodes to the 
Tanner graph of the LDPC code improves the LP decoder [2], 
[6]. However it is not desirable to add all such constraints as 
it will slow down the LP decoder considerably. 

In this work we propose an approach to adaptively add 
constraints to the LP decoder which, simultaneously, reduce 



the duality gap and are tractable (i.e., the number of such 
additional constraints are small and also each constraint in- 
volves only a small number of variables). Such approaches, 
which try to get rid off the fractional solution (or make the LP 
polytope tighter), have been used to improve the LP decoding 
of LDPC codes [2], [6]-[ll]. The new LP decoder which we 
propose, identifies frustrated cycles (see Section IIII-AI ) when 
the basic LP produces a fractional solution. We show that these 
frustrated cycles are the cause of inconsistency in the solution. 
Then we adaptively add them, as constraints, to the basic LP 
decoder This enables us to recover the transmitted codeword 
in many cases. We show empirically that the new LP decoder 
has an improved word error rate performance. Furthermore, 
the new LP decoder also has tractable complexity. 

II. Channel Model, Maximum Likelihood Decoder 
AND Linear Programming decoder 

A. Setup and Nomenclature 

We consider transmission over a BIAWGNC with noise 
distribution given by A/'(0,cr^). We use blocklength n LDPC 
encoding and denote x_ = {xi, X2, . . . , a;„} as the transmit- 
ted codeword. The input codebit takes value in {0, 1}. The 
received message is denoted by y G M". We will use the log- 
likelihood ratio (LLR) to represent the channel observations. 
More precisely, we have li = log p"'^!^' where Py\x{y\x) is 
the channel transition pdf. Let | represent the vector of LLRs. 

The LDPC code is represented by the usual Tanner graph 
representation[l]. Throughout the paper we will use {di,dr)- 
regular LDPC code ensembles to demonstrate our approach. 
The design rate of the LDPC code is given by \ — di/d^. 
In the experiments we perform later, we consider the random 
(3, 4)-regular LDPC code ensemble and the fixed 155-Tanner 
code [12] which has degree 3 variable nodes and degree 5 
check nodes. We use V to denote the set of n variable nodes 
or codebits and C to denote the set of m parity check nodes. 
A generic variable node and a check node is denoted by the 
letter i and c respectively. Let C represent the code (or the set 
of codewords). 

B. ML Decoder 

The ML decoder can be written as the following combi- 
natorial optimization problem [2], 'avmxec'}2l=i Uxi- This is 
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also the Integer Program (IP) representing the ML decoding. 

C. Basic Linear Programming Decoder 

For every check node c G C, let Xc = {xi | i £ c}. We also 
use c\i X.Q denote the set of all variable nodes contained in 
check node c except for the variable node i. The above IP can 
be relaxed to 

n 

inin^ X! hxibi{xi) 

~ i=l Xi£{0,l} 

s.t. VzeF: J2 = 

2;, £{0,1} 

Vc e C, Vi e c, Xi G {0, 1} : bi{xi) = ^ x^yi) 

Vc G C, Vxc s.t. a^i = 1, bc{xc) = 0, (local codeword) 

< b,{x,) <l, yieV, < < 1, Vc G C, 

which constitutes the standard LP decoder [2]. Here bi{xi) 
represents the "belief" of the variable node i and bc{xc) 
represents the "belief" associated to the check node c. In the 
sequel, we will also say that bi{xi) is the belief associated 
to the singleton clique i and bc{xc) is the belief associated 
to a higher order c//q'M43- Also, b represents the vector of 
all the variable node and check node beliefs. Note that the 
objective function represents the "cost" of decoding a bit to 0. 
This cost is reduced if the corresponding LLR is negative. The 
second condition imposed by the LP above is the consistency 
condition. In the third condition, the sum is over GF(2). 

III. Main Results: Improved LP decoding 

As mentioned earlier, our approach is to adaptively add con- 
straints to the LP which decrease the duality gap. Furthermore, 
we want the number of such additional constraints to be small 
and also each constraint to involve only a small number of 
variables. 

There are many existing approaches to improve the LP 
decoder [2], [6]-[ll]. In [2] an improved LP decoder based 
on "lift-and-project" method was introduced. In [8], the LP 
is enhanced by eliminating the facet containing the fractional 
solution. In [6], [7], extra constraints are added by combining 
parity checks which correspond to violated constraints to 
improve the LP performance. In [9] a mixed-integer LP was 
introduced by fixing the most "uncertain" bit of the pseu- 
docodeword. In [10] an adaptive LP decoder was introduced 
based on loop calculus. Critical loops were identified and 
then broken by fixing bits on the loop. In [11] a non- 
linear programming decoder was designed for decoding LDPC 
codes. 

A. LP Decoders using Frustrated Subgraphs 

Although our approach is in the same spirit as aforemen- 
tioned works, the main ideas are very different and have their 

'in a cirque, every node is connected to every other node. The LPs given 
in this paper always have beliefs associated to cliques. 



origins in [13] and [14]. Similar ideas have been independently 
used in [15], [16]. Before we describe the basic idea let us first 
define the notion of a frustrated graph. 

Definition 1 (Frustrated Graph): Consider a constraint sat- 
isfaction problem (CSP) defined on n binary (boolean) vari- 
ables, Xj, and m constraint nodes (each of which constraints 
a small set of variables). For each constraint c there are only 
certain configurations of Xc G {0, l}!"^' which satisfy it. Then, 
we say that the graph is frustrated if and only if there is 
no assignment of x which satisfies all m constraint nodes 
simultaneously. ■ 

Let us now define a CSP for our set-up. 

Definition 2 (CSP obtained by the LP Solution): Assume 
that the output of LP, b, is a fractional solution, i.e., we have 
a duality gap. For every clique c (with size at least two), 
the set of Xc which satisfy the clique are those for which 
bc{xc) > 0. In other words, the set of Xc satisfying the clique 
c, correspond to the support set of bc{xc). Consequently, 
the CSP is given by the 7i (binary) variables, {xi}"^^ and 
the set of cliques c (constraining the variables as described 
previously). ■ 

We now show that if the output of the LP has a frustrated 
subgraph, then it must have a duality gap, i.e., the solution 
must be fractional. 

Lemma 3: If there exists a frustrated subgraph, then there 
is a duality gap. 

Proof: Indeed, suppose on the contrary there was no 
duality gap, i.e, output of the LP is integral. Thus for every 
clique c (singleton or higher order), bc{xc) = 1 for some Xc G 
{0, l}!"^! and bc{xc) = for the rest. Consider any subset of the 
cliques, C = {ci,C2, . . . ,Cr}. Let x*. be such that baix*.) = 
1.0. We claim that U^^jX*. satisfies the CSP represented by 
C. Indeed, this follows from the consistency imposed by the 
LP. Thus no subgraph is frustrated. ■ 

Thus our strategy is as follows: first identify a frustrated 
subgraph from the output of the basic LP; if we add this 
frustrated subgraph as a constraint in our LP, then we ensure 
that this subgraph cannot be frustrated. In our experiments 
we see that, in many cases, adding the frustrated subgraphs 
eliminates the duality gap. 

To ensure that the subgraph we add as a constraint to the 
LP becomes consistent (or is not frustrated), we need to add 
all its maximal cliques and their intersections to the LP. More 
precisely, we add the maximal cliques of the junction tree|l of 
that subgraph as extra beliefs to the LP. 

The main challenge that remains is to find a frustrated 
subgraph in tractable time. In general, it is hard to find an 
arbitrary subgraph which is frustrated. We also remark that in 
[13] it was found empirically that the random field ising model 
could typically be solved (duality gap eliminated) by adding 
frustrated cycles arising in the LP solution. It is also known 
from Barahona's work (see references within [15]) that adding 
cycles is sufficient to solve the zero-field planar ising model. 
Hence as a first step, we focus on finding frustrated cycles of 

- See [13] for a discussion on Junction trees. It can be shown that 
running LP on the junction tree of a graph is optimal (equal to the original 
combinatorial optimization problem). If frustrated subgraph is a cycle then 
we just add all the tiiangles which chordalizes the cycle. 
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the graph. Frustrated cycles and a procedure to find them are 
described in the next section. The procedure is tractable and 
uses the implication graph method (to solve 2SAT problem) 
of [13], [17]. For details see Appendix B in [13]. 

B. Implication Graph and Frustrated Cycles 

For every clique c, consider all the two-projections 
of its belief. I.e., for every hc{xc), consider all the 
bij{xi, Xj) \/ i, j e c. These are obtained by summing 
out the other variables. We construct the implication graph as 
follows. In the implication graph each node i is present as 
(for Xi = 0) and i_ (for Xi = 1). Thus, the implication graph 
has a total of 2n nodes. There is a directed edge present 
between i and j which represents the logical implication 
obtained from bij{xi, Xj). Let us explain this in more details. 
To generate the logical implication, consider the set T of 
configurations of {xi,Xj) which render bij{xi,Xj) > and 
can introduce inconsistency. Thus, T is any of the following 
(01, 10), (01, 10, 11), (01, 10, 00), (00, 11), (00, 11, 10) and 
(00,11,01). Indeed, moments thought shows that other 
configurations, e.g., (00,01,10,11), are not restrictive 
and hence do not form any logical implication. Also, 
nodes which have integer beliefs are present as isolated 
nodes in the graph and do not have any edges entering 
or leaving it. Draw the directed edges using this 
T. E.g., suppose that LP outputs beliefs such that 
&,j(0,l) > 0,&,j(l,0) > 0,6,y(l,l) > 0,6,,,(0,0) = 
then T — (01,10,11). This implies a directed edge from 
i+ j- and j+ i_, because if 2;^ = then we must have 



Xj = 1 and if Xj = then a-^ = 1. In figure [T| we illustrate 
all possible implications which form the building blocks for 
constructing the implication graph. 
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Fig. 1. Figure shows all possible implications between Xi and Xj. These 
are used as basic building blocks to create the implication graph. 

Finally, a frustrated cycle is defined to be a directed cycle 
or a directed path which visits both and i_, once, for any 
i. One can find all such cycles and paths in a time which is 
linear in the number of nodes of the implication graph. 

Figure |2] shows the possible frustrated cycles which are 
obtained from the implication graph. The figure on the left 
shows true frustration. I.e., from the logical implications. 



obtained by the LP solution, we have that Xj = implies 
Xi = 1 and Xi = I implies Xi = 0. This means that the 
set of local beliefs (which lie on the cycle connecting i+ 
to i_) are not consistent. Hence it naturally suggests that 
there is frustration in the LP solution. The other kind of 
frustration, suggested by the remaining figures, is called as 
quasi-frustration. The figure in the middle demonstrates that 
Xi ~ 1 implies that 2;^ = but not the other way around. This 
quasi-frustration implies that there cannot be a global joint 
distribution (on all the variable nodes) such that it is consistent 
with the local beliefs. Indeed, if it were true, then we know 
that it must assign bi{xi = 0) > and bi{xi = 1) > 0. This is 
because the variable node i is present in the implication graph 
and hence must have a fractional solution for bi{xi). However, 
from the implication graph Xi = 1 implies Xi ~ 0, hence any 
configuration (on all nodes), which has a non-zero probability, 
cannot have Xi — 1, i.e., bi[x\ = 1) = 0, a contradiction. We 






Fig. 2. Figure shows the possible frustrated cycles present in the implication 
graph. The first cycle is truly frustrated, since we must have Xi = implies 
Xi = \ and vice-versa. The remaining two cycles are quasi-frustrated, since 
either = implies = 1 or vice-versa, but not both at the same time. 
These cycles are added to the LP and the enhanced decoder is termed LP- 
Frastrated Cycles (LP-FC). 

remark here that once we have found a frustrated cycle on 
the implication graph, one can easily obtain the cycle on the 
original graph, by just projecting the nodes on the implication 
graph back to the nodes on the original graph. The method in 
which we add the frustrated cycle to the LP is illustrated in 
the example below. 

Example 4 (Triangulation of Frustrated Cycles): Figure |3] 
shows a cycle {xi,X2,X3,X4,X5,XQ,xr,xs) which we add 
to the LP as a constraint. Adding the entire belief, 
b{xi,X2, ■ ■ ■ ,xs), as a constraint, would be expensive 
and result in 2® extra variables and constraints amongst 
them. Instead we add the maximal cliques of its junc- 
tion tree. To do this, we first chordalize or triangulate 
the cycle, as shown in the figure |3] into the 6 triangles 
given by (xi, a;2, X3), (xi, ^3, 0:4), (a;i, X4, a;5), (xi, 2:5, xg), 
{xi,XQ,xr), {xi,xr,xs)- These triangles are the maximal 
cliques and we add them as constraints to the LP. E.g., we 
add bx^x^^^{xi,X2,X3) for all xi,X2,X3 e {0,1}. For every 
belief that we add to the LP, we add constraints to ensure 
consistency with previously added beliefs. E.g., when we add 
{xi,X2,X3) and {xi,X3,X4)we introduce the 

constraint J2x2^^u^2,x3{^i^^3) = J2xi^xuX3,xi{xi,X3,X4) 
for all values of xi,X3. In other words, every clique that we 
add to the LP, must be consistent across its intersections. 

C. Experiments using Frustrated Cycles 

We consider BIAWGNC where the standard deviation 
of the noise is denoted by tr. We consider two types of 
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Fig. 3. Figure shows triangulation of the cycle 

{xt,X2,X3,X4,,X5,XQ,X7,xs,xi). The triangles chordalize the cycle 
and form the maximal cUques. The details are explained in example |4] 



LDPC encoding; (i) regular (3, 4) LDPC ensemble with 
design rate equal to 1/4 and the (ii) 155-Tanner code [12]. 
The 155-Tanner code has a design rate of 2/5. We let the 
standard deviation of the noise, a, take values in the set 
{0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.00, 1.05, 1.10, 1.15, 1.20}. We 
run 2000 trials for each value of a. We run experiments for 
both the (3, 4)-regular ensemble and the 155-Tanner code. For 
the (3, 4)-regular ensemble, in each trial a code is generated 
uniformly at random and used for transmission. 

Since we are transmitting over a symmetric channel, for the 
purpose of performance analysis we can assume that we are 
transmitting the all-zero codeword [1]. Under this assumption, 
the distribution of the LLRs are given by AA(-t-,^)- The 
generated LLRs are fed to both basic LP and LP-FC decoder 
The LP-FC algorithm is described below. For any decoder, if 
the output equals the all-zero codeword, we declare success, 
else there is an error We plot the word error rate ( WER) versu s 
the SNR (Eb/No) in dB. 

LP-FC Decoder: 

1) Run the basic LP. Go to step 4. 

2) If the output is fractional, find the frustrated 
cycle (FC) of the smallest length and add all its 
triangles. 

3) Rerun the LP 

4) If output is integral, stop else go to 2. 



1) Experiments with {i,4)-regular LDPC ensemble: Fig- 
ure |4] shows the performance curve when we use the (3,4)- 
regular ensemble with blocklength 160. The dark curve rep- 
resents the performance (averaged over 2000 trials where in 
each trial a code and noise realization is picked uniformly at 
random) when we use the basic LP decoder The gray curve 
denotes the performance under LP-FC. We remark here that for 
each simulation trial, the LP and LP-FC were run on the same 
code and noise realization. We observe that there were many 
trials where the basic LP decoder failed. However, adding a 
small number of cycles to the LP helped in retrieving the 
transmitted all-zero codeword. From the figures we observe 
that LP-FC performs much better than the basic LP. 

Table U demonstrates various quantities for different values 
of the SNR for the case when we use the (3, 4)-regular LDPC 
ensemble with blocklength 160. The second column shows the 
average number of LPs called in the LP-FC algorithm, i.e, the 
number of times step 3 is called in the LP-FC algorithm. The 
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Fig. 4. The figure shows the performance improvement of LP-FC over the 
basic LP. In this expeiiment (3, 4)-regular LDPC ensemble of blocklength 160 
was used. The dark curve depicts the word error rate (WER) performance of 
the basic LP and the gray curve shows the performance of the LP-FC. 
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TABLE I 

Complexity comparison of LP and LP-FC decoders. 



remaining columns illustrate the complexity of the extra LPs 
which are solved in the LP-FC algorithm. The third and the 
sixth column show the number of non-zeros in the constraint 
matrix and the dimensions of the constraint matrix when the 
basic LP is run. The fourth and the last column show the 
average number of non-zero entries in the constraint matrix 
and the average dimensions of the of the constraint matrix, 
when the LP-FC algorithm is run, respectively. Also shown in 
the fifth column is the maximum number of non-zero entries 
in any constraint matrix which occurs in the LP-FC algorithm. 
Thus, the table demonstrates that the size of the LP, after 
adding the frustrated cycles, does not increase by much. Hence 
the LP-FC decoder is kept tractable. 

We also observe that every cycle we add is a simple cycle, 
without any self-intersections. 

2} Experiments with 155-Tanner code [12]: We also per- 
form experiments with the 155-Tanner code which has 155 
variable nodes and 93 check nodes. The experimental set-up 
is same as before. 

Figure |5] shows the performance curve (averaged over 2000 
noise realizations for each value of a) when we use the 155- 
Tanner code. Again, we observe that LP-FC performs much 
better than the basic LP. 

We also perform experiments at very high SNR for the 155- 
Tanner code. This known as the error-floor regime. The error- 
floor occurs because of low-weight pseudocodewords which 
are fractional, i.e., not codewords. In [18] a pseudocodeword 
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Fig. 5. Tlie figure siiows tlie performance improvement of LP-FC over the 
basic LP wiien the 155-Tanner code was used. 

search algorithm was used to generate pseudocodewords which 
are not codewords. We pick 200 worst pseudocodewords 
which have effective weight [4] less than the minimum Ham- 
ming distance of 20. Also, all these pseudocodewords will 
dominate the WER when SNR becomes very large. 

The experiment we perform is as follows. We take the 
corresponding noise realizations which gave rise to these 200 
pseudocodewords. We run the basic LP on then and confirm 
that it fails on all these noise realizations and indeed we 
recover the fractional pseudocodewords. On the same noise 
realizations, we also run the LP-FC. Remarkably, the LP- 
FC is able to recover the correct (all-zero) codeword for all 
the 200 worst-case noise realizations. Furthermore, the step 3 
in the LP-FC algorithm was just called once. The constraint 
matrix for the basic LP is has 51,646 non-zeros entries and a 
dimension of (8618, 7006). On the other hand the enhanced LP 
has, on an average, 52,676 non-zeros entries and an average 
dimension of (8925, 7163). Again, the LP-FC is kept tractable. 

IV. Discussion 

In this work we present an improved LP decoder, called LP- 
FC, based on frustrated cycles. We show that the presence of 
frustration in the output of the basic LP solution is the cause 
of inconsistency. We add these frustrated cycles as constraints 
to the LP, thus enhancing it. We observe empirically that the 
LP-FC decoder eliminates the duality gap, in a large number 
of cases. Our simulations demonstrate that the LP-FC has a 
much better performance compared to the basic LP introduced 
in [2]. 

This approach toward enhancing the basic LP decoder opens 
up many interesting research directions. One direction is to 
investigate if one can add a frustrated subgraph, which is not 
a cycle, to enhance the LP, when the addition of cycles is not 
enough to eliminate the duality gap. The reason we choose to 
add frustrated cycles, is that as mentioned in Section IIII-BI 
the algorithm for finding such cycles is simple. It is not clear 
if there exists simple algorithms to find minimal frustrated 
subgraphs. 

Recently, improved LP detectors based on frustrated cycles 
was also used in [19] for 2DISI channel. One future re- 
search direction is to investigate other combinatorial problems 



in graphical coding, e.g., minimum pseudocodeword weight 
problem, minimum Hamming distance etc. 

Another future direction would be to develop distributed, 
i.e., message-passing, versions for the LP-FC. 
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